Arsen Apostolov writes about the actual electrical cost of running local Large Language Models on a single NVIDIA RTX 3090 compared to hosted cloud APIs.
>"I measured the actual GPU electricity for eight local models on one RTX 3090 — and the cheapest wasn't the smallest, nor the priciest the biggest"
Cost of Generating 1 Million Tokens Locally
| MODEL | PARAMS (Billions) | MEAN SPEED (tok/s) | AVG GPU DRAW (W) | € / 1M OUTPUT TOKENS |
| :--- | :---: | :---: | :---: | :---: |
| **gemma3:1b** | 1B | 136 tok/s | 154 W | €0.060 |
| **Qwen3-Coder** | 30.5B | 130 tok/s | 233 W | €0.112 |
| **gemma4:26b** | 26B | 85 tok/s | 246 W | €0.139 |
| **Devstral** | 24B | 49 tok/s | 320 W | €0.321 |
| **gemma3:27b** | 27B | 36 tok/s | 283 W | €0.361 |
| **Seed-OSS** | 36B | 4.5 tok/s | 186 W | €0.946 |
| **GLM-4.5-Air** | 106B | 5.7 tok/s | 141 W | €1.040 |
| **DeepSeek-R1-Distill** | 32.8B | 6.9 tok/s | 155 W | €1.526 |
By measuring real-time GPU power consumption through a custom dashboard, he discovered that token costs are driven by effective wall-clock throughput rather than model parameter size or raw generation speed alone. The results show that while small and fast models can be more economical than cloud services, reasoning-heavy models may actually become the most expensive to run locally due to the time spent "deliberating" between tokens.
* Measurements were performed using HomeLab Monitor, an open-source dashboard that integrates live power data from `nvidia-smi`.
* DeepSeek-R1-Distill emerged as the most expensive model per million tokens because its effective throughput is slowed by reasoning delays.
* The findings focus on marginal electricity costs and exclude total cost of ownership factors like hardware amortization or idle draw.
>"I measured the actual GPU electricity for eight local models on one RTX 3090 — and the cheapest wasn't the smallest, nor the priciest the biggest"
Cost of Generating 1 Million Tokens Locally
| MODEL | PARAMS (Billions) | MEAN SPEED (tok/s) | AVG GPU DRAW (W) | € / 1M OUTPUT TOKENS |
| :--- | :---: | :---: | :---: | :---: |
| **gemma3:1b** | 1B | 136 tok/s | 154 W | €0.060 |
| **Qwen3-Coder** | 30.5B | 130 tok/s | 233 W | €0.112 |
| **gemma4:26b** | 26B | 85 tok/s | 246 W | €0.139 |
| **Devstral** | 24B | 49 tok/s | 320 W | €0.321 |
| **gemma3:27b** | 27B | 36 tok/s | 283 W | €0.361 |
| **Seed-OSS** | 36B | 4.5 tok/s | 186 W | €0.946 |
| **GLM-4.5-Air** | 106B | 5.7 tok/s | 141 W | €1.040 |
| **DeepSeek-R1-Distill** | 32.8B | 6.9 tok/s | 155 W | €1.526 |
By measuring real-time GPU power consumption through a custom dashboard, he discovered that token costs are driven by effective wall-clock throughput rather than model parameter size or raw generation speed alone. The results show that while small and fast models can be more economical than cloud services, reasoning-heavy models may actually become the most expensive to run locally due to the time spent "deliberating" between tokens.
* Measurements were performed using HomeLab Monitor, an open-source dashboard that integrates live power data from `nvidia-smi`.
* DeepSeek-R1-Distill emerged as the most expensive model per million tokens because its effective throughput is slowed by reasoning delays.
* The findings focus on marginal electricity costs and exclude total cost of ownership factors like hardware amortization or idle draw.
The article explores how modern local large language models are evolving beyond mere quantization into unique architectures that outperform larger cloud-based counterparts in specific tasks. Rather than being simple smaller versions of existing systems, these new releases employ specialized training and attention mechanisms to handle context management, reasoning, and multimodality on consumer hardware efficiently.
Key developments include:
- Zaya1's use of compressed convolutional attention for efficient long-context reasoning.
- VibeThinker-3B focusing on dense mathematical and code intelligence in small models.
- DeepSeek V4 Flash leveraging sparse attention to run massive MoE architectures locally.
- Qwen 3.6 employing linear attention to maintain fixed context memory size.
- DiffusionGemma's non-autoregressive, parallel text generation via diffusion processes.
- Gemma 4 offering efficient on-device multimodal capabilities for mobile devices.
Key developments include:
- Zaya1's use of compressed convolutional attention for efficient long-context reasoning.
- VibeThinker-3B focusing on dense mathematical and code intelligence in small models.
- DeepSeek V4 Flash leveraging sparse attention to run massive MoE architectures locally.
- Qwen 3.6 employing linear attention to maintain fixed context memory size.
- DiffusionGemma's non-autoregressive, parallel text generation via diffusion processes.
- Gemma 4 offering efficient on-device multimodal capabilities for mobile devices.
The article discusses the benefits of running Google’s Gemma 4 models locally on personal hardware to ensure data privacy and independence from cloud services. By executing these multimodal models on a laptop, users can process images and audio without exposing sensitive information to third-party servers. The text highlights how efficient architecture allows for vision capabilities and speech recognition even with limited VRAM.
* Localized processing avoids the risks of uploading private or regulated data
* Native audio support in specific model variants like E2B and E4B
* Large context windows allow for deep analysis of lengthy documents and codebases
* Reduced reliance on internet connectivity during mobile workflows
* Localized processing avoids the risks of uploading private or regulated data
* Native audio support in specific model variants like E2B and E4B
* Large context windows allow for deep analysis of lengthy documents and codebases
* Reduced reliance on internet connectivity during mobile workflows
The Bonsai Demo repository provides a streamlined way to run Bonsai language models locally on various platforms, including macOS via Metal and Linux or Windows via CUDA. It offers support for multiple model sizes—8B, 4B, and 1.7B—in both GGUF and MLX formats, making it highly versatile for different hardware setups. The repository includes automated setup scripts that manage dependencies, Python environments, and model downloads from HuggingFace. Users can perform inference through command-line tools, start a built-in chat server, or even integrate with Open WebUI for a more interactive experience. This project is specifically optimized for efficient, high-performance local execution on Apple Silicon and CUDA-enabled GPUs.
A command-line tool that lets you interact with your terminal in natural language. All inference stays local.
